Skip to content

Conversation

MitchTurner
Copy link
Member

Linked Issues/PRs

followup to feedback from #3100

Description

Checklist

  • Breaking changes are clearly marked as such in the PR description and changelog
  • New behavior is reflected in tests
  • The specification matches the implemented behavior (link update PR if changes are needed)

Before requesting review

  • I have reviewed the code myself
  • I have created follow-up issues caused by this PR and linked them here

After merging, notify other teams

[Add or remove entries as needed]

@MitchTurner MitchTurner changed the title Use Proto Types in storage isn't of bytes Use Proto Types in storage instead of bytes Oct 6, 2025
@MitchTurner MitchTurner changed the base branch from master to chore/integrate-block-aggregator October 6, 2025 18:06
message BlockResponse {
oneof payload {
Block literal = 1;
string remote = 2;
Copy link
Member

@mchristopher mchristopher Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think remote should be an object so it's more flexible going forward. My initial thought it that it would look like this:

message RemoteBlockResponse {
  string path = 1;
  oneof config {
    RemoteBlockConfigS3 s3 = 2;
    RemoteBlockConfigHTTP http = 3;
  }
}

message RemoteBlockConfigS3 {
  string region = 1;
  string bucket = 2;
  bool requester_pays = 3;
  optional endpoint = 4;
}

message RemoteBlockConfigHTTP {
  string endpoint = 1;
  optional map<string, string> headers = 2;
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. Makes sense to me.

@MitchTurner MitchTurner marked this pull request as ready for review October 8, 2025 17:21
@MitchTurner MitchTurner requested review from a team, Dentosal, rymnc and xgreenx as code owners October 8, 2025 17:21
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

tokio::sync::mpsc::channel::<Result<BlockResponse, Status>>(16);
let (tx, rx) = tokio::sync::mpsc::channel::<
Result<ProtoBlockResponse, Status>,
>(16);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

16? is this arbitrary? if the client thats receiving the blocks is really slow, wouldn't that result in send failures from the spawned task?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll add better documentation.

if the client thats receiving the blocks is really slow, wouldn't that result in send failures from the spawned task?

I think the task will just block until the stream is cleared. From the okio::sync::mpsc docs:

    /// # Examples
    ///
    /// In the following example, each call to `send` will block until the
    /// previously sent value was received.
    ///
    /// ```rust
    /// use tokio::sync::mpsc;
    ///
    /// #[tokio::main]
    /// async fn main() {
    ///     let (tx, mut rx) = mpsc::channel(1);
    ///
    ///     tokio::spawn(async move {
    ///         for i in 0..10 {
    ///             if let Err(_) = tx.send(i).await {
    ///                 println!("receiver dropped");
    ///                 return;
    ///             }
    ///         }
    ///     });
    ///
    ///     while let Some(i) = rx.recv().await {
    ///         println!("got = {}", i);
    ///     }
    /// }
    /// ```
    ```
    
    I'm not exactly sure what the Tonic stream is doing under the hood. It may be clearing the buffer into its own buffer for us 🤷 

Comment on lines 147 to 152
ScriptTx script = 1;
// CreateTx create = 2;
// MintTx mint = 3;
// UpgradeTx upgrade = 4;
// UploadTx upload = 5;
// BlobTx blob = 6;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of these being ...Tx can we spell out Transaction? It keeps it cleaner & had no impact to the traffic on the wire.

Copy link
Collaborator

@xgreenx xgreenx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to verify the deserilziation of the blocks

Comment on lines +145 to +154
message Transaction {
oneof variant {
ScriptTransaction script = 1;
// CreateTx create = 2;
// MintTx mint = 3;
// UpgradeTx upgrade = 4;
// UploadTx upload = 5;
// BlobTx blob = 6;
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a todo please? And create an issues to implement it. Or do you want to do that before we merge to the master?

Copy link
Member Author

@MitchTurner MitchTurner Oct 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working on it here:
#3116

I can add an issue to be thorough.

type OwnedKey = BlockHeight;
type Value = Self::OwnedValue;
type OwnedValue = Block;
type OwnedValue = ProtoBlock;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ProtoBlock we want to use protobuf-based encoder and decoder, not Postcard

Comment on lines +52 to +53
// TODO: Should this be owned to begin with?
let (header, txs) = block.clone().into_inner();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should work with reference, cloning block looks like an overkill.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants